Question 1 [5 Pts]¶
What is Unsupervised Learning and difficulties involved in unsupervied Learning and name a few unsupervised algorithms ?
Solution¶
Unsupervised learning is a branch of machine learning which deals with unknown data that comes with a set of features and has no associated target variable. That is to say, there exists no prior knowledge about the outcome of modelling. Unsupervised learning focuses on dividing the data into groups or clusters such that it lays an understanding of the similarity/dissimilarity between different data points rather than predicting any outcome. It is a measure of interesting associations between different features such as determining subgroups and their intra cluster behaviour; determinng the differences between two groups or subgroups by measuring the inter cluster distances (any Euclidean or relevant distance measures).
Unsupervised learning comes with a drawback - not having a target variable. Now, if we understand that unsupervised learning is all about subjective analysis of data, it needs extensive study of data by different visulaization techniques and the statistical tools available at our disposal. This task becomes very difficult when there is no gaurantee or check measure available to verify the analysis. Imagine describing an unknown data and not having an idea of how good is your description or analysis. That is because there is no standardization of desired outcomes rather it is based on assumptions that are drawn out of visualization and statistical techniques applied to the data. That means, we will always need to do great hard work in describing the data because the analysis is based on few assumptions which may or may not be true.
Few unsupervised learning algorithms are - Principal Component Analysis, K-means clustering, KNN clustering, Hirarchical clustering.
Question 2 [5 Pts]¶
What is a PCA [Principal Component Analysis] and When to use PCA [Principal Component Analysis] ?
Solution¶
PCA is an unsupervised approach that is used to compute the principal components for a given dataset and the components are subsequently used for understanding the data. Principal components are the directions or vectors in a feature space along which the original data has most of the variance.
Imagine we are presented with a dataset with some hundreds of feature vectors, could that slow down or infact confuse our machine learning algorithm ? Yes, indeed, it will be a difficult task to handle so many features while traning the model. What if the features in the dataset have a relationship with the observations such that one feature is more dominant in decribing the data as compared to other features in the dataset? That means we can use the feature which decribes the data almost clearly (almost because it usually does not decribe 100% of the outcome) and avoid the ones which have least significance in the decision. This will help to reduce the overhead on the model. This task of summarizing the dataset with a smaller number of representative variables that collectively explain most of the variability in the original dataset is termed as principal component analysis.
Question 3 [5 Pts]¶
How does a PCA work [Principal Component Analysis] ?
Solution¶
PCA finds a low dimensional space for the given dataset with as much variance as possible. The idea is that each of the n observations lives in p-dimensional space, but not all of these dimensions are equally interesting. The measure of intrestingness is the amount of variation each observation has in the different dimensions. If one particular dimension preserves almost 70% of the varaiance in data which is the highest for that data on the given dimension, the same dimension can be used to represent the observations with 70% of the varaince. This dimension would be the principal component or the eigen vector corresponding to the largest eigen value of the data. It is also the first principal component of the data.
PCA works by calculating the average measurements of the data with respect to features and finds a centre point. Now the data is shifted such that the centre lies on top of the origin(0,0) in the coordinate plane. Lets call this center 'C'. Relative positions of the data points are always preserved. Now, we fit a line that passes through the point 'C'. The line is rotated until the appropriate position of the line that fits best to the data is found. Once the line is found, the projected data points are the representation of the observations on the given component.
PCA works by taking projections of data points on a given component and then it either measures the distances from the data to the line and tries to find the line that minimizes those distances or it can try to find the line that maximizes the distances from the projected points to the origin.
Now lets look at the maths behind the same. The first principal component of a set of features X1, X2,...,Xp is the normalized linear combination of the features that has the largest variance.
(Ref: An Introduction to Statistical Learning, Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani)
Z1 = φ11 X1 + φ21 X2 + ... + φp1 Xp
By normalized, we mean that
Summation over square(φj1) = 1
We refer to the elements φ11,...,φp1 as the loadings of the first principal component. The loadings make up the principal component loading vector, φ1 = (φ11 φ21 ... φp1)T .
The aim is to maximize the objective function that is defined above. This set of loading vectors and the maximization of the equation presented above can be found by eigen value decomposition. That helps in finding all the eigen values for a given data along with their corresponding eigen vectors such that the largest eigen vector corresponds to the largest eigen value. This largest eigen vector is the first principal component that retains the maximum possible variance on a single dimension of the dataset.
Question 4 [5 Pts]¶
What are different methods by which you can compute the PCA? Does every method will yield same result or answer at the end?
Solution¶
There exist two different ways that can be used to compute PCA:
1) Eigen value Decomposition¶
The PCs can be determined via eigen decomposition of the covariance matrix C (X*X.transpose) of the data matrix X. Eigen decomposition, in fact, finds a new coordinate system of the eigenvectors for C through various rotations.
C = E D E.transpose¶
Matrix of eigenvectors E( m x m) and a diagonal matrix of "m" eigenvalues D is the eigen decomposition of C (m x m). The eigenvectors or column vectors in E are the PCs we wish to evaluate. We then use matrix multiplication the data projection onto the principal components.
2) Singular Value Decomposition¶
Consider the same representation of covariance matrix stated above. "CC.transpose" is U and "C.transposeC" is V such that U and V are orthogonal matrices, i.e. "VV.transpose = I(indentity matrix)" and "UU.transpose = I(indentity matrix)".
By SVD rule:
C = U S V.Transpose¶
where S matrix is a compositon of singular values found by application of square root onto the eigen values of U and V such that all the singular values are arranged diagonally in descending order.
Yes, the aim of both the methods is the same, i.e. to find the data in the directions with the highest variances. Yet, SVD is considered more stable and suited for complex, square or rectangular matrices of data while eigen decomposition can only work with square matrices.
Question 5 [5 Pts]¶
What are advantages and disadvantages of PCA[Explain with example]?
Solution¶
Advantage of PCA is that it reduces the higher dimensions of data to lower dimensions such that the most of the variance is preserved and the overhead of compuation for a machine learning algorithm decreases with the reduction in dimensionality of data. Usually, the information in a data set is concentrated in its first few principal components and noise, if any, tend towards the later principal components. The same is applicable when we wish to visualize the relation between different features, it becomes highly impossible to visualize and infer from data that has high dimensions. PCA helps in visualizing the higher dimensional data in lower dimensions. Consider a motor insurance fraud detection system which tries to predict how claims made by different customers could be fradulent. In this the dataset may contain, name of the customer, details about previous claims, amount claimed, marital status, injury type of the vehicle, and gender, etc., amongst which few features say like marital status or gender may not be of great importance in the prediction. PCA could help reduce the dimensions of data such that relevant predictions are made.
Disadvantage of PCA is that it highly relies on the approximation of points, i.e. the projection of data points in lower dimensions that result in some loss of information in data. That means the original independent features are no more existing rather the principal components have created a new derived features from the different correlated features. Standardization is a compulsory step before PCA as the features with high values can influence the variance of data on the respective components.
Question 6 [10 Pts]¶
What is clustering? Explain how K-Means Clustering Algorithm works?
Solution¶
Clustering is an unsupervised way of grouping data into different groups. Observations in same subgroups share some similarity amongst the observations within them and are different from the observations from different subgroups.
K-means clustering works by dividing the dataset into non-overlapping subgroups such that the intra cluster distances are minimized and the inter cluster distances are maximized.
1 - The number of clusters are choosen.
2 - Each cluster must contain one centroid. Centroid is the mean value of the observations in a cluster i.e the centre point of the set of observations in a cluster.
3 - Initially, since the clusters are not known, centroids are assigned at random.
4 - Then each observation's distance from all the centroids is evaluated. The observation is assigned to a cluster for which the centroid has the least value of the distance to the given observation
5 - After all the observations are assigned to clusters, mean value of the cluster is evaluated. The new centroid is updated to the mean value
6 - Then the steps 4 and 5 are followed again until the algorithm converges to a local minimum. i.e. when the centroids are no more changing. This is the point when the clusters are defined with clear boundaries.
Question 7 [10 Pts]¶
What are the Advantages and disadvantages of Clustering ALgorithms discussed in our class (K-Means,Hierchal)?
Solution¶
Advantages of hierarchical and k-means is that it is relatively simple to understand and implement and guarantees convergence. The dendrogram produced after applying hierarchical clustering can be used to understand the bigger picture along with the subgroups of the data. Clustering is very useful in detecting the subgroups among the data.
In k-means clustering deciding upon K even before starting the clustering is the main disadvantage for k-means. In k-means or hirarchichal clustering, the given observation or data point can belong to only one cluster i.e. even if the data might share similarity with other clusters, the clustering algorithm constraints the set of points to be at only one cluster and not within multiple clusters.
Question 8 [5 Pts]¶
Which Clustering Algorithm is better K-Means or hierarchical Clustering? Explain with a proper example which is better algorithm?
Solution¶
Hirarchical clustering is any time better than K-means, the biggest factor being choice of cluster number must be given even before the k-means algorithm starts training. Unlike k-means, hirarchical clustering doesnt need a mention of clusters before the process starts. For example consider a dataset of covid-19 outbreak that consists the details of outbreak across the globe. We intend to study the most affected cluster groups and their impact on the spread of desease across the globe. K-means demands to give the required number of clusters before, that in a way is a problem because our intention is to cluster the data based on the simmilarity or dissimilarity measure within it without any prior knowledge of how many clusters should or could be present. In such case, hirarchical clustering helps in not only finding the subgroups without the mention of clusters initially, but also defines a tree like structure i.e. a dendogram.
Question 9 & 10 [ 50 Pts]¶
Please Perform Principal Component Analysis and K-Means Clustering on the Give dataset Below.
10 Points for Data Preprocessing.
15 Points for PCA Algorithm along with plots and Results Explaination.
15 Points for K-Means Algorithm with plots and Results Explination.
10 Points for Comparing the results between PCA and K-Means and whats your inference from your ouputs of the algorithms.
Hints:
As per the data preprocessing step convert all the variables in the dataset into Numerical values as the algorithms only work with Numerical values
Then Apply both algorithms one after the other then plot the ouput clusters
Compare the output clusters in both the steps.
Bank Marketing Data Set¶
URL: https://dataminingcsc6740.s3-us-west-2.amazonaws.com/datasets/homework_2.csv
Files Information:¶
the homework_2.csv has all the informatiom.
Relevant Information:¶
The data is related with direct marketing campaigns of a Portuguese banking institution. The marketing campaigns were based on phone calls. Often, more than one contact to the same client was required, in order to access if the product (bank term deposit) would be (or not) subscribed.
There are two datasets: 1) The csv file has 4521 data points in it. 2) Number of Attributes: 16 + output attribute.
Attribute information:¶
Input variables:
bank client data:¶
1 - age (numeric)
2 - job : type of job (categorical: "admin.","unknown","unemployed","management","housemaid","entrepreneur","student", "blue-collar","self-employed","retired","technician","services")
3 - marital : marital status (categorical: "married","divorced","single"; note: "divorced" means divorced or widowed)
4 - education (categorical: "unknown","secondary","primary","tertiary")
5 - default: has credit in default? (binary: "yes","no")
6 - balance: average yearly balance, in euros (numeric)
7 - housing: has housing loan? (binary: "yes","no")
8 - loan: has personal loan? (binary: "yes","no")
related with the last contact of the current campaign:¶
9 - contact: contact communication type (categorical: "unknown","telephone","cellular")
10 - day: last contact day of the month (numeric)
11 - month: last contact month of year (categorical: "jan", "feb", "mar", ..., "nov", "dec")
12 - duration: last contact duration, in seconds (numeric)
other attributes:¶
13 - campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)
14 - pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric, -1 means client was not previously contacted)
15 - previous: number of contacts performed before this campaign and for this client (numeric)
16 - poutcome: outcome of the previous marketing campaign (categorical: "unknown","other","failure","success")
Output variable (desired target):
17 - y - has the client subscribed a term deposit? (binary: "yes","no")
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns
data = pd.read_csv('homework_2.csv',delimiter=';')
data.head()
Made a copy of the pandas dataframe such that the categorical varaibles align one after other. This helps in preprocessing the data easily and I do not want to manipulate the original data that was imported along with the target variable. Since unsupervised learning deals with only features of the data without any prior information on the target variable, I seperate the target variable from the dataset.
x = pd.DataFrame(index=range(data.shape[0]), columns=['age','balance','day','duration','campaign','pdays','previous','job', 'marital', 'education', 'default', 'housing',
'loan', 'contact', 'poutcome','month'])
x = pd.DataFrame(data, dtype=None, copy=False, index = x.index, columns=x.columns )
y = pd.DataFrame(data, dtype=None, copy=False, index = x.index, columns=['y'] )
x.head()
x.info()
No null values in the dataset¶
x.isnull().sum()
There are no outliers in the dataset¶
print(x.loc[x['age']<18])
print(x.loc[x['day']<0])
print(x.loc[x['duration']<0])
print(x.loc[x['campaign']<0])
print(x.loc[x['pdays']<-1])
print(x.loc[x['previous']<0])
Descriptive Statistics for the dataset¶
x.describe()
Encoding¶
The categorical columns in the dataset should be changed to numeric values for further processing of data using PCA or K-Means
x['job'] = x['job'].astype('category')
x['marital'] = x['marital'].astype('category')
x['education'] = x['education'].astype('category')
x['default'] = x['default'].astype('category')
x['housing'] = x['housing'].astype('category')
x['loan'] = x['loan'].astype('category')
x['contact'] = x['contact'].astype('category')
x['poutcome'] = x['poutcome'].astype('category')
x['month'] = x['month'].astype('category')
x['job'] = x['job'].cat.codes
x['marital'] = x['marital'].cat.codes
x['education'] = x['education'].cat.codes
x['default'] = x['default'].cat.codes
x['housing'] = x['housing'].cat.codes
x['loan'] = x['loan'].cat.codes
x['contact'] = x['contact'].cat.codes
x['poutcome'] = x['poutcome'].cat.codes
x['month'] = x['month'].cat.codes
x.dtypes
x.head()
Scaling is important before doing PCA¶
from sklearn.preprocessing import StandardScaler
#Feature Scaling using StandardScaler
x_scaled = StandardScaler().fit_transform(x)
x_scaled = pd.DataFrame(x_scaled, dtype=None, copy=False, index = x.index, columns=x.columns)
x_scaled.head()
from sklearn.decomposition import PCA
pca = PCA()
principal = pca.fit_transform(x_scaled)
principal_component = pd.DataFrame(pca.components_.T , index = x_scaled.columns
, columns = ['PC1', 'PC2','PC3','PC4','PC5','PC6','PC7','PC8','PC9','PC10','PC11','PC12','PC13','PC14','PC15','PC16'])
principalDf = pd.DataFrame(principal, index = x_scaled.index, columns = principal_component.columns)
principalDf
principal_component
Visualization of Data on Principal Components¶
Lets visualize the datapoints along different combination of PCs such that a scatter plot can describe the 2-dimensinal spread of the data across the principal components.
import seaborn as sns
sns.set()
sns.pairplot(principalDf,diag_kind='hist', height = 2.0)
plt.show()
By looking at the scatter plot above, we see that the data points are too closely represented in every combination. This shows how just 1 or 2 PCs won't be enough to represent the entire information in the data. There will be a great loss of variance if only 2 PC's are to be considered.
PCs Variance¶
Now that we have created our required principal componenets, we must look at the variance values of the PCs so that we get to decide how many PCs actually account for the best cumulative preservation of the information (variance) provided by the data.
principal_variance = pd.DataFrame(pca.explained_variance_ratio_*100 , columns = ['% Variance Preserved']
, index = ['PC1', 'PC2','PC3','PC4','PC5','PC6','PC7','PC8','PC9','PC10','PC11','PC12','PC13','PC14','PC15','PC16'])
principal_variance['Cumulative % Variance'] = principal_variance['% Variance Preserved'].cumsum()
principal_variance
fig, ax = plt.subplots(1,2,squeeze=False,constrained_layout=True,figsize=(16,3))
ax[0][0].bar(principal_variance.index,principal_variance['% Variance Preserved'])
ax[0][0].set_title('% Variance Preserved')
ax[0][1].bar(principal_variance.index,principal_variance['Cumulative % Variance'])
ax[0][1].set_title('Cumulative % Variance Preserved')
plt.show()
Find best PCs¶
Now, looking at the bar graphs mentioned above and the elbow graphs presented below, we can see that PC1-PC11(inclusive) the variance percentage is over 80 and thus, the 16 dimensional data can be represented using these 11 principal components such that the maximum variance is preserved without losing much on the data.
However, the choice of cumulative variance depends on the business requirement. The choice should be made such that the cumulative variance crosses the required threshold (For eg 80% and above).
Here for understanding purpose, I mentioned PC1-PC11 will be a good choice.
plt.figure(figsize=(13,6))
plt.plot(principal_variance.index, principal_variance['% Variance Preserved'], '-o')
plt.plot(principal_variance.index, principal_variance['Cumulative % Variance'], '-s')
plt.xticks(principal_variance.index)
plt.ylabel('Percentage of Variance Explained')
plt.xlabel('Principal Component')
plt.legend(['% Variance Preserved','Cumulative % Variance Preserved'])
plt.title('Variance Graph for PCs')
plt.show()
Let us look at an example which obtains the first 11 PCs that preserve over 80% variance in data¶
pca_ex = PCA(n_components = 11)
principal_ex = pca_ex.fit_transform(x_scaled)
principal_component_ex = pd.DataFrame(pca_ex.components_.T , index = x_scaled.columns , columns = ['PC1', 'PC2','PC3','PC4','PC5','PC6','PC7','PC8','PC9','PC10','PC11'])
principalDf_ex = pd.DataFrame(principal_ex, index = x_scaled.index, columns = principal_component_ex.columns)
principalDf_ex
This summarizes the Principal component analysis for dimensionality reduction
K-Means¶
Visualizing correlation between features¶
The correlation between different variables(features) can be best explained using heatmaps. The same is plotted below. A lighter shade represents high correlation and darker shade represents less correlation.
cor = x_scaled.corr()
sns.heatmap(cor, square = True) #Plot the correlation as heat map
from sklearn.cluster import KMeans
def computeKMeans(samples, n=2,max_iter=300):
km = KMeans(n_clusters = n,max_iter=max_iter)
km.fit(samples)
clusters = km.predict(samples)
centroids = km.cluster_centers_
inertia = km.inertia_
score = km.fit(samples).score(samples)
return (clusters, score, inertia)
iner = []
sc = []
for i in range(1,15):
clusters, score, inertia = computeKMeans(x_scaled, n=i, max_iter=400)
iner.append(inertia)
sc.append(score)
plt.figure(figsize=(13,6))
plt.plot(range(1,15),iner)
plt.plot(range(1,15),sc)
plt.title('Elbow Method')
plt.xlabel('Number of clusters')
plt.legend(['Inertia','Score'])
plt.show()
Choice of number of clusters based on the elbow graph¶
Based on the above-mentioned elbow graph analysis that explains the inertia with respect to the number of clusters, it is evident that the k-means algorithm's rate of decrease in inertia (Sum of squared distances of samples to their closest cluster center) decreases as the number of clusters increase beyond 6. For example, cluster number 2 has approximately 63000 inertia value and 4 has 53000, i.e. by an increase of 2 clusters, inertia has reduced by 10000 which is a significant amount as compared to when the cluster number is changed from 6 to 8 (reduced by 4000 approx).
Hence, we can choose 6 clusters to represent the data such that the inertia is minimum and this helps in defining the clusters well.
clusters, score, inertia = computeKMeans(x_scaled, n=6, max_iter=400)
kmeans = pd.DataFrame(clusters)
x_kmeans = pd.DataFrame(x_scaled).copy()
x_kmeans['kmeans clusters'] = kmeans
inertia
x_kmeans.head()
Visualization of features with respect to the clusters¶
After the k-means clustering is performed, we add an additional column to the x_scaled dataframe such that the column values represent the respective clusters to which the given row of data belongs to. This will help us in plotting the datapoints on an SPLOM. The following visualization helps to see the distribution of data points with respect to one feature vs the other along with the clusters defined by the color of the points. Each subplot represents distribution across two features plotted using x and y axis.
import seaborn as sns
sns.set()
sns.pairplot(x_kmeans,hue='kmeans clusters',diag_kind='hist', height = 3.0)
plt.show()
Hirarchical clustering¶
Lets plot the dendogram structure of the x_scaled dataset using scipy (https://docs.scipy.org/doc/scipy/reference/cluster.hierarchy.html). The y-axis represents the Euclidean distance and the x-axis are the data points (observations). First we determine the linkages between data points and then plot the dendogram
import scipy.cluster.hierarchy as h
plt.figure(figsize=(10, 7))
plt.title("Dendrograms")
sh = h.linkage(x_scaled, method='ward')
dendo = h.dendrogram(sh)
sh
Now that we have visualized the dendogram, we can choose the threshold at which we would like to cut through the dendogram such that the vertical linkages are broken and the dendogram is divided into different clusters. By visual cue, I decide upon 55 on y-axis a good measure to divide in clusters. Hence, the number of clusters will be 8.
plt.figure(figsize=(10, 7))
plt.title("Dendrograms")
dendo = h.dendrogram(sh)
plt.axhline(y=55, color='r', linestyle='--')
After visualizing the dendogram and choosing clusters we can perform agglomerative clustering which is one type of hirarchical clustering. The same can be done using AgglomerativeClustering library provided by sklearn.
from sklearn.cluster import AgglomerativeClustering
hc = AgglomerativeClustering(n_clusters = 8, affinity = 'euclidean', linkage ='ward')
clusters_hc = hc.fit_predict(x_scaled)
clusters_hc
hc_x = pd.DataFrame(x_scaled)
hc_x['hc_cluster'] = pd.DataFrame(clusters_hc).copy()
hc_x.head()
import seaborn as sns
sns.set()
sns.pairplot(hc_x,hue='hc_cluster',diag_kind='hist', height = 3.0)
plt.show()
Inference and Comparision of K-means, PCA and Hirarchical clustering¶
PCA looks to find a low-dimensional representation of the observations that explain a good fraction of the variance whereas Clustering looks to find homogeneous subgroups among the observations.
The PCA algorithm is best suited for dimensionality reduction of a dataset before performing any supervised machine learning algorithms for high dimensional dataset. However, as we can see in the elbow or bar plots presented for PCA that it helps in determinig the components that best represent the data and yet can not preserve 100% of variance or information in the data if choosen even 1 less than the maximum number of components available. That is to say, we reduce the dimensions by compromising on some information in the data. PCA works by taking projections of data points on a given component and then it either measures the distances from the data to the line and tries to find the line that minimizes those distances or it can try to find the line that maximizes the distances from the projected points to the origin.
K-means on the other hand uses data points to centroid distance (Euclidean or any distance) measure to evaluate or group datasets into different clusters. This is not specifically losing up on data unlike PCA which projects data points onto different components (eigen vectors) to summarize the variablilty of data in the best posisble way. K-means focuses on cluster analysis rather than component analysis which results in grouping of data into relevant clusters. Choice of clusters depends on the factor such as how inter cluster distances are maximized and intra cluster distances are minimized.
Hirarchical clustering could be assumed as a better version than k-means where we do not have to worry on choosing the clusters before fitting the data to the model. Once the data is fit using hirarchical clustering we can use the threshold on Euclidean distance such that it can divide the complete tree or dendogram into individual clusters
Based on the analysis of the visualizations presented in this document, clustering using hirarchical is more intuitive as compared to the k-means. While I agree, k-means suits best for few scenarios where the data is quite dispersed as in to identify well defined clusters by a simple visualization of the spread. But, in cases where the data is high dimensional and is extremely overlapping i.e. a simple scatter plot visualization will never help to decide the clusters rather techniques like elbow method for evaluating inertia are choosen, I woud prefer hirarchical clustering where the visual cue is more easy to understand via dendograms.